Analytical Study of the Interplay between Architecture and Predictability

We study model feed forward networks as time series predictors in the stationary limit The focus is on complex yet non chaotic behavior The main question we address is whether the asymptotic behavior is governed by the architecture regardless the details of the weights We nd hierarchies among classes of architectures with respect to the attractor dimension of the long term sequence they are capable of generating larger number of hidden units can generate higher dimensional attractors In the case of a perceptron we develop the stationary solution for general weights and show that the ow is typically one dimensional The relaxation time from an arbitrary initial condition to the stationary solution is found to scale linearly with the size of the network In multilayer networks the number of hidden units gives bounds on the number and dimension of the possible attractors We conclude that long term prediction in the non chaotic regime with such models is governed by attractor dynamics related to the architecture Neural networks provide an important tool as model free estimators for the solution of problems when the real model is unknown or weakly known In the last decade there has been a growing interest in the application of such tools in the area of time series prediction see Weigand and Gershenfeld In this paper we analyse a typical class of architectures used in this eld i e a feed forward network governed by the following dynamic rule S Sout S t j S t j j N where Sout is the network s output at time step t and S t j are the inputs at that time N is the size of the delayed input vector The rational behind using time delayed vectors as inputs is the theory of state space reconstruction of a dynamic system using delay coordinates Takens Sauer Yorke and Casdagli This the ory address the problem of reproducing a set of states associated with the dynamic system using vectors obtained from the measured time series and is widely used for time series analysis A similar architecture incorporating time delays is the TDNN time delay neural network with a recurrent loop Waibel et al This type of networks is known to be appropriate for learning temporal sequences e g speech signal In the context of time series it is mostly used for short term predictions Our analysis focuses on the various long time properties of the sequence generated by a given architecture and the interplay between them The aim of such an investiga tion is the understanding and characterization of the long term sequences generated by such architectures and the time scale to reach this asymptotic behavior Such knowledge is necessary to de ne adequate measures for the transition between a locally dependent prediction and the long term behavior Though some work has been done on characterization of a dynamic system from its time series using neu ral networks not much analytical results that connect architecture and long time prediction are available see M Mozer in Weigand and Gershenfeld Nev ertheless practical considerations for choosing the architecture were investigated extensively Weigand and Gershenfeld and references therein It has been shown that such networks are capable of generating chaotic like sequences While it is possible to reconstruct approximately the phase space of chaotic attractors at least in low dimension it is clear that prediction of chaotic sequences is limited by the very nature of such systems namely the divergence of the distance between nearby trajectories Therefore one can only speak about short time predictions with respect to such systems Our focus is the ability to generate complex sequences and the relation between architecture and the dimension of such sequences